Methods And Systems For Managing A Data Mining Model

ABSTRACT

Methods, systems, and a computer readable medium storing a computer executable program for managing a data mining model are disclosed. A first notification associated with a data mining model is received at a business process system. A second notification associated with the data mining model is transmitted from the business process system responsive to the first notification. A determination is made regarding whether a first response operable to define an association between the data mining model and business data has been received at the business process system responsive to the second notification. A command is issued from the business process system to update model metadata associated with the data mining model based on the determination.

FIELD OF THE INVENTION

The present invention generally relates data mining models and moreparticularly to methods and systems for managing a data mining model.

BACKGROUND OF THE INVENTION

Businesses routinely use business intelligence systems that involve theuse of data mining models to make sense of increasingly large volumes ofbusiness data including but not limited to, for example, marketing data.A business intelligence system may at times include over a hundredoperational data mining models. Business decisions made by businessanalysts are often based on predictions generated by such data miningmodels. In some cases, individual business rules are developed bybusiness analysts for implementation based on specific predictionsgenerated by one or more data mining models.

Effectively running a business intelligence system typically involvescoordination between at least two different parties, the modeladministrators and the business analysts. The model administrators mayinclude, but are not limited to, model developers, model experts, andstatisticians. The model administrators develop customized data miningmodels, add new models to the system, monitor model performance, and/orperform model maintenance. The business analysts rely on the data miningmodels to make business decisions including those involving the designand updating of business rules. Business analysts typically like to beinformed of changes and updates to data mining models of relevance totheir business area so that they can synchronize their businessdecisions in accordance with the status of available data mining models.

Some prior art data mining model management systems, such as forexample, SAS Enterprise Miner, Microsoft Analysis Services, Oracle DataMining and Analytics, and Fairlsaac Model Builder provide data miningand model management platforms for model administrators. Such prior artdata mining management system fail to create a unified model managementframework that provides tools for facilitating interactions betweenmodel administrators and business analysts.

SUMMARY OF THE INVENTION

One aspect of the invention is directed to a method of managing a datamining model. A first notification associated with a data mining modelis received at a business process system. A second notificationassociated with the data mining model is transmitted from the businessprocess system responsive to the first notification. A determination ismade regarding whether a first response operable to define anassociation between the data mining model and business data has beenreceived at the business process system responsive to the secondnotification. A command is issued from the business process system toupdate model metadata associated with the data mining model based on thedetermination.

Another aspect of the invention is directed to a computer readablemedium storing a computer executable program for managing a data miningmodel. Yet another aspect of the invention is directed to a system formanaging a data mining model.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram representation of an example of a system thatmay be used to implement one embodiment of managing a data mining model;

FIG. 2 is a block diagram representation of one embodiment of a businessprocess system;

FIG. 3 is a state diagram representation of one embodiment of alifecycle of a data mining model;

FIG. 4 is a block diagram representation of one embodiment of datamining model events associated with a data mining model;

FIG. 5 is a representative example of basic model information in anexample of one embodiment of a model specific metadata associated with adata mining model;

FIG. 6 is a representative example of a model schema definition in anexample of one embodiment of model specific metadata associated with adata mining model;

FIG. 7 is a representative example of model assumptions in an example ofone embodiment of model specific metadata associated with a data miningmodel;

FIG. 8 is a representative example of model specific keywords in anexample of one embodiment of model specific metadata associated with adata mining model;

FIG. 9 is a representative example of model performance evaluationdefinition in an example of one embodiment of model specific metadataassociated with a data mining model;

FIG. 10 is representative example of model performance based eventtriggers of an example of one embodiment of model specific metadataassociated with a data mining model;

FIG. 11 is a representative example of model business rule dependenciesin an example of one embodiment of model specific metadata associatedwith a data mining model;

FIG. 12 is a representative example of inter-model relationships in anexample of one embodiment of model specific metadata associated with adata mining model;

FIG. 13 is a flowchart representation of one embodiment of a method ofhandling a Model Created Event at a business process system;

FIG. 14 a flowchart representation of one embodiment of a method ofhandling a Model Referenced Event at a business process system;

FIG. 15 is a flowchart representation of one embodiment of a method ofevaluating data mining models;

FIG. 16 is a flowchart representation of one embodiment of a method ofhandling a Model Deteriorated Event at a business process system;

FIG. 17 is a flowchart representation of one embodiment of a method ofhandling a Model Updated Event at a business process system;

FIG. 18 is a flowchart representation of one embodiment of a method ofhandling a Model To Be Expired Event at a business process system;

FIG. 19 is a flowchart representation of one embodiment of a method ofhandling a Model Unreferenced Event at a business process system; and

FIG. 20 is a flowchart representation of one embodiment of a method ofmanaging a data mining model.

DETAILED DESCRIPTION OF THE DRAWINGS

Referring to FIG. 1, a block diagram representation of an example of asystem 100 that may be used to implement one embodiment of data miningmodel management is shown. The system 100 generally includes a businessprocess system 102, a model developer system 104, a business analystsystem 106, a model metadata repository 108, a data mining modeldatabase 110, a model evaluation system 112, and a text parsing/matchingsystem 114. The business process system 102 is communicatively coupledto the model developer system 104, the business analyst system 106, themodel metadata repository 108, the model evaluation system 112, and thetext parsing/matching system 114. The model developer system 104 iscommunicatively coupled to the model metadata repository 108 and thedata mining model database 110. The model evaluation system 112 iscommunicatively coupled to the model metadata repository 108 and thedata mining model database 110.

In one embodiment, text parsing/matching system 114 is an integratedcomponent of the business process system 102. In one embodiment, themodel evaluation system 112 is an integrated component of the businessprocess system 102. In one embodiment, the model metadata repository 108is an integrated component of the data mining model database 110. Whilea single business analyst system 106 is shown, alternative embodimentsof the system 100 may include additional business analyst systems 106.While a single model developer system 104 is shown, alternativeembodiments of the system 100 may include multiple model developersystems 104. In one embodiment, model developer systems 104 include oneor more of model author systems, statistician systems, and model expertsystems.

The business process system 102 generally coordinates communicationsbetween the model developer system 104 and the business analyst system106. In one embodiment, the business process system 102 generallycoordinates the management of data mining models stored in the datamining model database 110. In one embodiment, the business processsystem 102 manages one or more of model deployment, model updates, modelbusiness relationships, model evaluation, model performance monitoring,inter-model dependency relationships, and model expiration. The modeldeveloper is generally responsible for the creation, deployment, andmaintenance of data mining models via the model developer station 104.

The business analyst is generally responsible for making businessdecisions based on predictions generated by data mining models. In oneembodiment, the business analyst authors business rules based onpredictions generated by data mining models via the business analystsystem 106. The business process system 102 provides the businessanalyst with data mining model related notification via the businessanalyst system 106 that inform business analysts of model deployments,model changes, model updates, model expiration, and prompts tosynchronize data models with business rules.

The data mining model database 110 stores one or more data mining modelsthat have been deployed for use in the system 100. In one embodiment,the data mining model database 110 only includes actively deployed datamining models. In one embodiment, the data mining model databaseincludes an archive of expired data mining models. The model evaluationsystem 112 runs one or more model evaluation routines to evaluate thestatus of a data mining model responsive to a request received from thebusiness process system 102. The text parsing/matching system 114 isused to compare model specific keywords with model keywords subscribedto by one or more business analysts.

The model metadata repository 108 is generally used to store modelspecific metadata associated with each of the data mining models. In oneembodiment, the model metadata repository 108 stores model specificmetadata associated with each of the data mining models stored in thedata mining database 110. In one embodiment, the model metadatarepository 108 stores model specific metadata associated with datamining models actively deployed in the system 100. In one embodiment,the model metadata repository 108 includes an archive of model specificmetadata associated with expired data mining models.

Referring to FIG. 2, a block diagram representation of one embodiment ofa business process system 102 is shown. The business process system 102generally includes a processing unit 200 communicatively coupled to amemory 204 and a communication module 206. The processing unit 200generally includes a processor or controller. The communication module206 generally coordinates the exchange of data between the businessprocess system 102 and the model developer system 104, the businessanalyst system 106, the model metadata repository 108, the modelevaluation system 112, and the text parsing/matching system 114.

The operating system module 208 and the data mining management module210 are stored in the memory 204. Alternative embodiments may includeadditional modules that facilitate the performance of data mining modelmanagement functions. In one embodiment, the memory 204 includes one ormore of a non-volatile memory, a volatile memory, and/or one or morestorage devices. Examples of non-volatile memory include, but are notlimited to, electrically erasable programmable read only memory (EEPROM)and read only memory (ROM). Examples of volatile memory include, but arenot limited to, static random access memory (SRAM), and dynamic randomaccess memory (DRAM). Examples of storage devices include, but are notlimited to, hard disk drives, compact disc drives, digital versatiledisc drives, and flash memory devices. The processing unit 200 generallyretrieves and executes machine readable instructions or softwareprograms that are stored in the memory 204.

Referring to FIG. 3, a state diagram representation of one embodiment ofa lifecycle of a data mining model 300 is shown. And FIG. 4 is a blockdiagram representation of one embodiment of data mining model events 400associated with the lifecycle of the data mining model detailed in FIG.3. The business process system 102 updates the model specific metadataassociated with the data mining model at the model metadata repository108 to record the history of the data mining model.

In one embodiment, the data mining model is created by a model developerand deployed for use via the model developer system 104. In oneembodiment, the data mining model is a previously created data miningmodule deployed for use in the system 100. The model developerdesignates the data mining model as a created model 302 at the modeldeveloper system 104 upon deployment of the data mining model for use bythe system 100. The model developer raises a Model Created Event 402 viathe model developer system 104 upon deployment of the data mining modelfor use in the system 100. The model developer system 104 communicates aModel Created Event notification to the business process system 102.

The business process system 102 communicates a Model Creatednotification to one or more business analysts via the business analystsystem 106 responsive to the Model Created Event. The one or morebusiness analysts are provided with the option of creating associationsbetween business data and the data mining model. In one embodiment, thebusiness analyst is provided with the option of developing businessrules dependent on predictions generated by the data mining model. If abusiness analyst creates an association between business data and thedata mining model, the data mining model is designated a referencedmodel 304. The business analyst raises a Model Referenced Event 404 uponthe creation of an association between business data and the data miningmodel at the business analyst system 106. The business analyst system106 communicates a Model Referenced Event notification to the businessprocess system 102.

The business process system 102 establishes a schedule based on modelmetadata for use in the model evaluation system 112. The modelevaluation system 112 executes evaluations according to the establishedschedule and issues commands to evaluate the data mining model. If themodel evaluation system 112 determines that the performance of the datamining model has deteriorated the model evaluation system designates thedata mining model as a deteriorated model 306. The deteriorated statusof the data mining model is communicated to the business process system102 and the business process system 102 responsively raises a ModelDeteriorated Event 406. The business process system 102 communicates aModel Deteriorated notification to one or more model developersassociated with the data mining model.

The model developer reviews the model evaluation results and determineswhether the data mining model can be fixed by updating one or more modelparameters. If the model developer determines that the data mining modelcan be fixed, the model developer updates the data mining model,designates the data mining model as an updated model 308 and raises aModel Updated Event 408 via the model developer system 104. The modeldeveloper system 104 communicates a Model Updated notification to thebusiness process system 102. The updated model 308 is periodicallyevaluated and if the updated model 308 is designated as a deterioratedmodel 306 again, the Model Deteriorated Event 406 is raised again andthe process for handling the deteriorated model 306 described above isrepeated.

If the model developer determines that the data mining model cannot befixed, the model developer designates the data mining model as a to beexpired model 310 and raises the Model To Be Expired Event 410 via themodel developer system 104. In one embodiment, the model developer setsa model expiration time for the data mining model. A Model To Be Expirednotification is communicated from the model developer system 104 to thebusiness process system 102. The business process system 102communicates a Model To Be Expired notification to one or more businessanalysts that have previously created associations between the datamining model and business data via the business analyst system 106.

Each of the business analysts removes previously created associationsbetween business data and the data mining model. In one embodiment, eachtime a business analyst removes an association between the data miningmodel and a business rule at the business analyst system 106, a ModelUnreferenced Event 412 is raised automatically by the business analystsystem 106. In one embodiment, the Model Unreferenced Event 412 israised manually by the business analyst. The business analyst system 106communicates a Model Unreferenced notification to the business processsystem 102. The business process system 102 sends a notification to oneor more business analysts that have previously created associationsbetween the data mining model and business data via the business analystsystem 106. Once the business process system 102 determines that all ofthe business rules that previously referenced the data mining model havebeen removed or unreferenced, the data mining model is designated anunreferenced model 312. In one embodiment, unreferenced models 312 areexpired automatically based on a pre-defined model expiration time. Inone embodiment, deteriorated models 306 are left in place for future usewith other datasets. In one embodiment, unreferenced models 312 are leftin place for future use with other datasets.

Once the business process system 102 determines that all of the businessrules that previously referenced the data mining model have been removedor unreferenced, the business process system 102 determines whether thedata mining model was designated as a to be expired model 310. If thebusiness process system 102 determines that the data mining model waspreviously designated as a to be expired model 310, the business processsystem 102 designates the data mining model as an expired model 314 andraises the Model Expired Event 414. The business process system 102communicates a Model Expired notification to the business analysts viathe business analyst system 106.

As mentioned previously, the model metadata repository 108 is generallyused to store model specific metadata associated with each of the datamining models. In one embodiment, the model specific metadata includesone or more of the following model parameters: basic model information,model schema definition, model assumptions, model specific keywords,model performance evaluation definition, model performance based eventtriggers, model business rule dependencies, and inter-modeldependencies.

In one embodiment, the basic model information includes one or more ofthe following basic model information data fields: a model identifier, amodel name, a model creation time, one or more model authors, a modeldescription, one or more data mining algorithm, a model last updatedtime, a model training set. The model identifier is a globally uniqueidentifier assigned to a data mining model. In one embodiment, the modelidentifier is a machine readable name. The model name is typically auser friendly name that is used by model developers, statisticians,model experts, and business analysts to reference the data mining model.The model creation time is a timestamp indicating the time at which thedata mining model was created. The model authors are the one or moremodel developers, statisticians, and model experts involved in thecreation and maintenance of the data mining model. The model descriptionis a textual description of the purpose of the data mining model. Thelisting of the one or more data mining algorithms identifies thespecific data mining algorithms used to construct the data mining model.The model last updated time is a timestamp that indicates the last timethat the data mining model was updated. The model training set is a setof database records based on a select statement stored in acorresponding relational statement in the model metadata repository 108.

Referring to FIG. 5, a representative example of basic model information500 associated with an example of one embodiment of a data mining modelis shown. The model identifier is 3596B3FB-7A50-4cf3-8A30-6AF0248E90C4.The model name is Personalized Coupon Offering. The model creation time306 is Mar. 14, 2005 12:25 PM. The model author is John Doe. The modeldescription defining the purpose of the data mining model to offercustomer in-store coupons based on past purchase history. The datamining algorithm used to construct the data mining model is DecisionTree. The model was last updated on May 17, 2005 2:55 PM. The modeltraining set used to train the data mining model is354A1338-E25C-4481-8F41-39D052823C18. It should be noted that while oneembodiment of basic model information associated with a data mining modehas been described, alternative embodiments of basic model informationthat include a subset of the basis model information data describedabove or that include other types of basic model information data beyondthose described above are also considered to be within the scope of theinvention.

In one embodiment, the model specific metadata includes a model schemadefinition. The model schema definition defines one or more inputattributes received at the data mining model and the output attributegenerated by the data mining model. In one embodiment, the inputattribute is an input parameter retrieved from a business database or abusiness data table. In one embodiment, each input parameter correspondsto a column in a database table. In one embodiment, the input attributeis an aggregated input parameter. An aggregated input parameter isderived from one or more input parameters retrieved from one or morebusiness databases or business data tables. In one embodiment, the inputattribute includes a combination of one or more input parameters and oneor more aggregated input parameters. In one embodiment, the outputattribute is a prediction generated by the data mining model based onthe input attributes.

In one embodiment, each input attribute includes one or more of thefollowing input attribute data fields: an input attribute name, an inputattribute data type, and an input attribute description. In oneembodiment, each output attribute includes one or more of the followingoutput attribute data fields: an output attribute name, an outputattribute data type, and an output attribute description. In oneembodiment, the input attribute description and the output attributedescription provide user friendly textual descriptions of the associatedinput attribute and the output attribute, respectively.

Referring to FIG. 6, a representative example of a model schemadefinition 600 associated with an example of one embodiment of a datamining model is shown. The model schema definition includes two inputattributes. The first input attribute, an input parameter, has an inputattribute name Customer, an input attribute data type String and aninput attribute description Customer identifier. The second inputattribute, an aggregated input parameter, has an input attribute namePurchasedLast3 Months, an input attribute data type Float and an inputattribute description Total purchases by customer from store during thelast three months. The output attribute has an output attribute nameTop10Coupon, an output attribute data type List <Integer> and an outputattribute description The top 10 personalized coupons to offer tocustomer from in-store kiosk based on customer purchasing history. Itshould be noted that while one embodiment of a model schema definitionhas been described above, alternative embodiments of model schemadefinition including alternative formats for input and/or outputattributes are also considered to be within the scope of the invention.

In one embodiment, the model specific metadata includes one or moremodel assumptions. A model developer typically develops a data miningmodel to operate under a pre-defined set of conditions. The pre-definedset of conditions is recorded in the associated model specific metadataas model assumptions. In one embodiment, the recorded model assumptionsare accessible to model developers, model experts, and statisticians viaa model developer system 104 to facilitate the evaluation, updatingand/or maintenance of the data mining model. In one embodiment, themodel assumptions are accessible to business analysts via a businessanalyst system 106 to facilitate the development of business rules basedon the data mining model.

In one embodiment, the model developer identifies an initial set ofmodel assumptions during the construction of a data mining model. In oneembodiment, the model developer updates the model assumptions during theupdating and maintenance of the data mining model. In one embodiment, amodel assumption is an input attribute model assumption associated withan input attribute defined in the model schema definition. In oneembodiment, a model assumption is an environmental model assumption thatdefines an environmental operating condition associated with the datamining model. In one embodiment, the model assumptions are expressed viaa labeling mechanism including a first order predicate.

Referring to FIG. 7, a representative example of model assumptions 700associated with an example of one embodiment of a data mining model isshown. A first model assumption Loyalty (Customer) >6 months is anexample of an input attribute model assumption involving the first inputattribute Customer defined in the model schema definition of FIG. 6. Theterm Loyalty is the name of the first order predicate of the first inputattribute Customer. The first input attribute Customer is filtered toinclude only those customers that have been shopping at the store forover six months. A second model assumption model assumption Total NumberOf In-Store Kiosks <=250 is an example of an environmental modelassumption. A third model assumption Product Category=Electronicsspecifies that the data mining model is designed to provide predictionsassociated with the purchase of electronics. A fourth model assumptionGeographical Region=North America specifies that the data mining modelis designed to provide predictions associated with purchases made inNorth America. It should be noted that while a number of different typesof model assumptions have been described above, alternative types ofmodel assumptions are also considered to be within the scope of theinvention.

In one embodiment, the model specific metadata includes one or moremodel specific keywords. Each data mining model is typically constructedwith one or more defined purposes. In one embodiment, the model specifickeywords are derived from the one or more defined purposes. In oneembodiment, the model specific keywords operate to semantically link twoor more of the data mining models. In one embodiment, the model specifickeywords are indexed and the data mining models are searchable by modelspecific keyword.

Referring to FIG. 8 a representative example of model specific keywords800 associated with an example of one embodiment of a data mining modelis shown. The output attribute description in the model schemadefinition of FIG. 6 describes the output attribute as The top 10personalized coupons to offer customer from an in-store kiosk based oncustomer purchasing history. The three model specific keywords: CouponOffering, In-Store Kiosk Offering, and Purchase Behavior Detection areconsistent with the description of the data mining model outputattribute. It should be noted that while a model specific keyword basedor model specific tag-based data mining model selection approach hasbeen described above, alternative data mining model querying approaches,such as for example, a relational query approach may also be usedwithout departing from the spirit of the invention.

In one embodiment, the model specific metadata includes a modelperformance evaluation definition. The model performance evaluationdefinition includes a listing of the evaluation routines used toevaluate the performance of the data mining model. In one embodiment,each listed evaluation routine includes one of more of the followingevaluation routine data fields: an evaluation routine assembly, anevaluation routine method, and an evaluation dataset. The evaluationroutine assembly is a fully qualified class name, which can be locatedin a predefined location, with a collection of static evaluationmethods. At runtime, once the evaluation routine assembly is loaded intothe running process's memory the corresponding evaluation method can beexecuted. The evaluation routine method defines the methodology employedby the evaluation routine. The evaluation routine dataset defines thedataset used by the evaluation routine to perform the data mining modelevaluation.

Referring to FIG. 9, a representative example of a model performanceevaluation definition 900 associated with an example of one embodimentof a data mining model is shown. The model performance evaluationdefinition 900 lists two evaluation routines to be used in theevaluation of the associated data mining model. The first evaluationroutine listed specifies the use of an evaluation routine assemblyDataMiningPackage, an evaluation routine method Compute ROC defined inthe evaluation routine assembly DataMiningPackage, and an evaluationdataset AE1284EB-1008-43b0-94AB-E922100BE68E. The second evaluationroutine listed specifies the use of an evaluation routine assemblyDataMiningPackage, an evaluation routine method Compute Retention Rate,and an evaluation dataset FD95B1DD-A77A-4c51-A0BA-03541EDE44F2.

In one embodiment, the model specific metadata includes one or moremodel performance based event triggers. Each evaluation routine listedin the model performance evaluation definition has a counterpartperformance based event trigger. After a data mining model has beenevaluated in accordance with the associated model performance evaluationdefinition, the results of the performance evaluation are analyzed inaccordance with the performance based event triggers to assess theperformance status of the data mining model. In the event that athreshold value defined in the performance based trigger is passed, aModel Deteriorated Event 406 is raised.

Each performance based event trigger includes one or more of thefollowing performance based event trigger data fields: an event name, anevaluation routine assembly, an evaluation routine method, an evaluationdataset, and a threshold value. In one embodiment, each performancebased event trigger includes an event name and a threshold value. In oneembodiment, each performance based event trigger includes a thresholdvalue.

The event name is a descriptive name of the performance based evaluationtrigger. The evaluation routine assembly and the evaluation routinedataset specified in the performance based event trigger are the same asthose specified in the counterpart evaluation routine. The evaluationroutine method specifies the methodology employed to determine whetherthe performance based event trigger has been triggered. The thresholdvalue defines a threshold value and a condition where if the conditionis found to be true, triggers the performance based event trigger. Ifthe performance based event trigger is triggered, the data mining modelis considered to be operating in a deteriorated state.

Referring to FIG. 10, a representative example of model performancebased event triggers 1000 of an example of one embodiment of a datamining model is shown. A first performance based event trigger isassociated with the first evaluation routine listed in the modelperformance evaluation definition of FIG. 9. The first performance basedevent trigger, ROC Decay Notification, specifies a threshold valueROC<0.57. The results of the first evaluation routine, Compute ROC, arecompared against the threshold value of 0.57. If the computed ROC isless than 0.57, the data mining model is considered to be in operatingin a deteriorated state.

A second performance based event trigger is associated with the secondevaluation routine listed in the model performance evaluation definitionof FIG. 9. The second performance based event trigger Customer RetentionRate Dropped specifies a threshold value Retention Rate<0.73. Theresults of the second evaluation routine, Compute Retention Rate, arecompared against the threshold value of 0.73. If the computed RetentionRate is less than 0.73, the data mining model is considered to beoperating in a deteriorated state. In one embodiment, retention rate isnot specific to particular data mining models therefore the data miningmodels to be deteriorated by this performance based trigger arespecified.

When a business analyst creates a business rule associated with a datamining model at a business analyst system 106, a globally uniquebusiness rule identifier is assigned to the newly created businessrules. The model identifier associated with the data mining model andthe business rule identifier associated with the newly created businessrule are communicated from the business analyst system 106 to thebusiness process system 102. The business process system 102 updates themodel business rule dependencies section of the model specific metadataassociated with the data mining model in the model metadata repository108.

Each business rule listed in the model business rule dependenciesincludes one or more of the following data fields: a business ruleidentifier and one or more business rule authors. The business ruleidentifier is the globally unique business rule identifier and thebusiness rule author(s) are the business analyst(s) responsible forcreating the business rule.

Referring to FIG. 11, a representative example of model business ruledependencies 1100 associated with an example of one embodiment of modelspecific metadata associated with a data mining model is shown. Themodel business rule dependencies associated with the data mining modelinclude two business rules. The first business rule lists the businessrule identifier as 636BFB24-5D83-4f4b-B8C1-FE4C732B653E and the businessrule author as Jane Smith. The second business rule lists the businessrule identifier as B88F949D-4802-4fd9-9BBB-3561C2120C37 and the businessrule author as Mike Jones.

In one embodiment, the model specific metadata includes inter-modelrelationships. In some cases, a data mining model may have aninter-model relationship with another data mining model. In oneembodiment, the inter-model relationship is a model improvedrelationship. In a model improved relationship, a first data miningmodel back-fits a second data mining model, by having more input data.In one embodiment, the inter-model relationship involves a first datamining model using an output generated by a second data mining model asan input. The first and second data mining models typically employdifferent data mining algorithms. As mentioned previously, each datamining model and each dataset has a globally unique model identifier.The inter-model relationships for a first data mining model includes themodel identifiers for each of the other data mining models that have aninter-model relationship with the first data mining model.

Referring to FIG. 12, a representative example of inter-modelrelationships 1200 in an example of one embodiment of model specificmetadata associated with a first data mining model is shown. Theinter-model relationships include two model identifiers associated withsecond and third data mining models that have an inter-modelrelationship with the first data mining model. The two model identifiersfor the second and third data mining models areA316EDC9-A250-438d-AED8-F37E08846B37 and82C1F8C6-F497-48a1-80D5-049FED00CcBEA, respectively.

In one embodiment upon the creation of a data mining model, the modeldeveloper transmits the newly created data mining model from the modeldeveloper system 104 to the data mining model database 110. In oneembodiment, the model developer transmits the newly created data miningmodel from an outside model developer system to the data mining modeldatabase 110. The model developer defines the model specific metadataassociated with the newly created data mining model and transmits themodel specific metadata to the model metadata repository 108. In oneembodiment, the model developer transmits the model specific metadatafrom the model developer system 110 to the model metadata repository108.

The model developer raises a Model Created Event associated with thenewly created data mining model indicating that the newly created datamining model is available for use by the system 100. In one embodiment,the Model Created Event is raised upon the addition of a newly addeddata mining model to the system 100. The model developer communicates aModel Created Event notification from the model developer system 104 tothe business process system 102. In one embodiment, the model developersystem 104 transmits the Model Created Event notification to thebusiness process system 102. In one embodiment, the model developersystem 104 transmits the Model Created Event notification to thebusiness process system 102 by invoking the web service method publishedby the business process system 102.

Referring to FIG. 13, a flowchart representation of one embodiment of amethod of handling a Model Created Event at the business process system102 is shown. In one embodiment, the business process system 102receives the Model Created Event notification at step 1302. The ModelCreated Event notification includes a Model Created Event payload. TheModel Created Event payload includes one or more types of basic modelinformation including a model identifier and a list of model specifickeywords associated with the newly added data mining model.

A business analyst is typically involved in specific areas of a businessand is therefore interested in data mining models associated with thespecific areas of the business. In one embodiment, the business analystsubscribes to selected model keywords relating to the specific areas ofthe business. The business process system 102 retrieves the modelspecific keywords from the Model Created Event payload at step 1304 andtransmits the retrieved model specific keywords to the textparsing/matching system 114 to identify the business analyst subscribersto the model specific keywords at step 1306.

The text parsing/matching system 114 performs a textual parsing of themodel specific keywords and a matching of the model specific keywords tobusiness analyst specific model keywords. The text parsing/matchingsystem 114 identifies the business analyst subscribers associated withthe model specific keywords based on identified matches between themodel specific keywords and the business analyst specific keywords. Thetext parsing/matching system 114 transmits the identities of thebusiness analyst subscribers to the business process system 102. Theidentities of the business analyst subscribers are received by thebusiness process system 102 at step 1308.

The business process system 102 communicates a Model Creatednotification associated with the newly added data mining model to thebusiness analyst subscribers identified by the text parsing/matchingsystem 114 at step 1310. In one embodiment, the business process system102 transmits the Model Created notification to the business analystsystem 106. In one embodiment, the business process system 102 transmitsthe Model Created notification to the business analyst system 106 byinvoking the web service method published by the business analyst system106.

Upon the receipt of the Model Created notification at the businessanalyst system 106, the business analyst inspects the data mining modeland if desired, develops one or more business rules associated with thedata mining model. The business analyst raises the Model ReferencedEvent at the business analyst system 106. In one embodiment, thebusiness analyst manually raises the Model Referenced Event at thebusiness analyst system 106. In one embodiment, the Model ReferencedEvent is automatically raised by the business analyst system 106 uponrecognition of the creation of a business rule associated with a datamining model.

The business analyst system 106 communicates a Model Referenced Eventnotification to the business process system 102. In one embodiment, thebusiness analyst system 106 transmits the Model Referenced Eventnotification to the business process system 102. In one embodiment, thebusiness analyst system 106 transmits the Model Referenced Eventnotification to the business process system 102 by invoking the webservice method published by the business process system 102.

Referring to FIG. 14, a flowchart representation of one embodiment of amethod 1400 of handling a Model Referenced Event at the business processsystem 102 is shown. The business process system 102 receives the ModelReferenced Event notification at step 1402. The Model Referenced Eventnotification includes a Model Referenced Event payload. In oneembodiment, the Model Referenced Event payload includes a modelidentifier, one or more business rule identifiers, and one or morebusiness analyst identifiers of business analysts responsible forauthoring each of the business rules.

The business process system 102 retrieves the model identifier, thebusiness rule identifier(s), and the business analyst identifier(s) fromModel Referenced Event payload at step 1404. The business process system102 transmits the retrieved model identifier and the business ruleidentifier(s) and the business analyst identifier(s) for the businessanalysts responsible for authoring each of the business rules to themodel metadata repository 108 for storage at step 1406.

The model metadata repository 108 identifies the model specific metadataassociated with the received model identifier. The model metadatarepository 108 stores the received business rule identifier(s) andassociated business analyst identifier(s) in the model business ruledependencies section of the identified model specific metadata. Whenfuture model events arise with respect to the data mining modelassociated with the business rules, the business analysts responsiblefor authoring the business rules are notified of such model events bythe business process system 102.

Referring to FIG. 15, a flowchart representation of one embodiment of amethod 1500 of evaluating data mining models is shown. The modelevaluation system 112 initiates an evaluation of the data mining modelsdeployed in the system 100. In one embodiment, the model evaluationsystem 112 initiates the evaluation process based on an evaluationschedule established by the business process system 102. The businessprocess system 102 establishes a schedule based on model metadata foruse by the model evaluation system 112. The model evaluation system 112executes evaluations according to the established schedule and issuescommands to evaluate the data mining model. In one embodiment, the modelevaluation system 112 initiates the evaluation process responsive to thereceived event based trigger. In one embodiment, the business processsystem 102 receives an event based trigger. The received event basedtrigger is communicated to the model evaluation system 112. In oneembodiment, the model evaluation system 112 initiates the evaluationprocess responsive to a web service call.

The model evaluation system 112 retrieves selected model specificmetadata associated with a first data mining model from the modelmetadata repository 108 at step 1502. The retrieved model specificmetadata includes an evaluation policy parameter. The model evaluationsystem 112 reviews the evaluation policy parameter and determineswhether the first data mining model is scheduled for evaluation at step1504. If the model evaluation system 112 determines that the first datamining model is not scheduled for evaluation, the model evaluationsystem 112 returns to step 1502 and retrieves the selected modelspecific metadata for the next data mining model from the model metadatarepository 108. If the model evaluation system 112 determines that thefirst data mining model is scheduled for evaluation, the modelevaluation system 112 evaluates the first data mining model at step1506.

The model evaluation system 112 retrieves selected model specificmetadata from the model metadata repository 108 including the modelperformance evaluation definition. The model performance evaluationdefinition includes a listing of the evaluation routines used toevaluate the data mining model. As mentioned previously each evaluationroutine listing includes an evaluation routine assembly, an evaluationroutine method, and an evaluation dataset. The model evaluation system112 runs each of the listed evaluation routines and populates a modelperformance table for with the performance results generated by themodel evaluation routines for the first data mining model.

The model evaluation system 112 retrieves selected model specificmetadata from the model metadata repository 108 including the modelperformance based triggers. Each performance based event triggerincludes a threshold value. The model evaluation system 112 compareseach of the performance results against the defined threshold values. Ifany of the conditions defined by the threshold values in the modelperformance based triggers are found to be TRUE, the first data miningmodel is designated as a deteriorated model. If the conditions definedby the threshold values of the model performance based triggers are allfound to be FALSE, the performance status of the first data mining modelremains designated as an operational model.

A determination is made at step 1508 regarding whether the first datamining model has deteriorated. If the data mining model remainsdesignated as an operational, the method 1500 proceeds to step 1512. Inone embodiment, if the first data mining model has been designated asdeteriorated by the model evaluation system 112, the Model DeterioratedEvent 406 is raised by the model evaluation system 112 at step 1510 anda Model Deteriorated Event notification is communicated to the businessprocess system 102. The method 1500 proceeds to step 1512. In oneembodiment, if the first data mining model has been designated asdeteriorated by the model evaluation system 112, the deteriorated statusof the first data mining model is communicated to the business processsystem 102 and the business process system 102 responsively raises theModel Deteriorated Event 406 at step 1510. The method 1500 proceeds tostep 1512.

At step 1512, a determination is made regarding whether there are anydata mining models awaiting evaluation. In one embodiment, the modelevaluation system 112 determines whether there are any additional datamining models awaiting evaluation at step 1512. In one embodiment, thebusiness process system 102 determines whether there any additional datamining models are awaiting evaluation at step 1512. If there are datamining models awaiting evaluation, the method returns to step 1502 andthe model evaluation process is repeated for the next data mining model.If there are no data mining models awaiting evaluation, the modelevaluation is considered to be complete at step 1514.

In one embodiment, one particular way used to structure and define theperformance evaluation methods and evaluation criteria is defining astatic function in a library for each performance evaluation. Theevaluation dataset used by the evaluation routine is specified for eachperformance evaluation. The model evaluation system 112 retrieves eachperformance evaluation at runtime. The performance evaluation resultsare stored in a model performance table for future performanceinspection. The business process system 102 scans through theperformance evaluation results, compares each of the performance resultsagainst the threshold values and raises a Model Deteriorated Event forthe data mining model if the performance criteria evaluated as TRUE. Theperformance evaluation result is stored in a dataset and is specified aspart of the event routine specification.

Referring to FIG. 16, a flowchart representation of one embodiment of amethod 1600 of handling a Model Deteriorated Event at the businessprocess system 102 is shown. The business process system 102 recognizesa Model Deteriorated Event for a data mining model at step 1602. Thebusiness process system 102 retrieves selected model specific metadataassociated with the data mining model including the model author(s) ofthe data mining model from the model metadata repository 108 at step1604. In one embodiment, the model authors include the modeldeveloper(s), the model expert(s) and/or the statistician(s) involved inthe creation and/or maintenance of the data mining model.

The business process system 102 communicates a Model Deterioratednotification associated with the deteriorated data mining model to theone or more identified model authors at step 1606. In one embodiment,the business process system 102 transmits the Model Deteriorated Eventnotification to the model developer system 104. In one embodiment, thebusiness process system 102 transmits the Model Deteriorated Eventnotification to the model developer system 104 by invoking the webservice method published by the model developer system 104.

In one embodiment, the Model Deteriorated Event notification includes amodel performance access link. The model performance access linkprovides the model author(s) with access to the model performanceresults generated by the model evaluation system 112 for thedeteriorated data mining model. In one embodiment, the model performanceaccess link is a button. In one embodiment, the model performance accesslink is a URL.

The model author determines whether the deteriorated data mining modelcan be fixed by updating one or more model parameters such as forexample, including but not limited to, one or more internal modelparameters. If the model author determines that the deteriorated datamining model can be fixed by updating one or more model parameters, themodel author updates the model parameters, designates the data miningmodel as an updated model, and raises a Model Updated Event at the modeldeveloper system 104.

A Model Updated Event notification is communicated from the modeldeveloper system 104 to the business process system 102. In oneembodiment, the model developer system 104 transmits the Model UpdatedEvent notification to the business process system 102. In oneembodiment, the model developer system 104 transmits the Model UpdatedEvent notification to the business process system 102 by invoking theweb service method published by the business process system 102.

Referring to FIG. 17, a flowchart representation of one embodiment of amethod 1700 of handling a Model Updated Event at the business processsystem 102 is shown. The Model Updated Event notification is received atthe business process system 102 at step 1702. The business processsystem 102 issues a command to the model metadata repository 108 toupdate the model specific metadata associated data mining model toreflect that the data mining model has been updated at step 1704. In oneembodiment, the business process system 102 issues a command to updatethe model last updated field of the basic model information section ofthe model specific metadata.

In some cases, one of several model authors involved in the creationand/or maintenance of the data mining model may have performed the modelupdates. In one embodiment, all of the model authors associated with thedata mining model are informed that the data mining model has beenupdated. The model author information is stored in the basic modelinformation section of the model specific metadata. The business processsystem 102 retrieves the model author information the model metadatarepository 108 at step 1706. In one embodiment, the model authorsinclude the model developer(s), the model expert(s) and/or thestatistician(s) involved in the creation and/or maintenance of the datamining model.

The business process system 102 communicates a Model Updatednotification associated with the updated data mining model to the modelauthors at step 1708. In one embodiment, the business process system 102transmits the Model Updated notification to the model developer system104. In one embodiment, the business process system 102 transmits theModel Updated notification to the model developer system 104 by invokingthe web service method published by the model developer system 104.

If the model author determines that the deteriorated data mining modelcannot be fixed, the model author designates the data mining model as ato be expired data mining model and raises a Model To Be Expired Eventat the model developer system 104. In one embodiment, a data miningmodel is considered unfixable if the terms, products and/or conceptsassociated with the data mining model have become obsolete. In oneembodiment, the model author sets a model expiration time for thedeteriorated data mining model.

A Model To Be Expired Event notification is communicated from the modeldeveloper system 104 to the business process system 102. In oneembodiment, the Model To Be Expired Event notification is transmittedfrom the model developer system 104 to the business process system 102.In one embodiment, the model developer system 104 invokes the webservice method published by the business process system 102 andtransmits the Model To Be Expired Event notification via this webservice invocation.

Referring to FIG. 18, a flowchart representation of one embodiment of amethod 1800 of handling a Model To Be Expired Event at the businessprocess system 102 is shown. The Model To Be Expired Event notificationis received at the business process system 102 at step 1802. Thebusiness process system 102 retrieves the model business ruledependencies section of the model specific metadata associated with thedeteriorated data mining model at step 1804. As mentioned previously,each business rule listed in the model business rule dependenciesincludes a business rule identifier and a business rule author. Thebusiness rule identifier is the globally unique business rule identifierand the business rule author is the business analyst responsible forcreating the business rule.

The business process system 102 communicates a Model To Be Expirednotification associated with the deteriorated data mining model to eachof the business analysts listed in the model business rule dependenciessection at step 1806. In one embodiment, the Model To Be Expirednotification includes the model expiration time for the to be expiredmodel. In one embodiment, the Model To Be Expired Event notification istransmitted from the business process system 102 to the business analystsystem 106. In one embodiment, the business process system 102 invokesthe web service method published by the business analyst system 106 andtransmits the Model To Be Expired Event notification via the web serviceinvocation.

When the business analysts receive the Model To Be Expired notification,each business analyst removes the associations between the businessrules authored by the business analyst and the to be expired data miningmodel. When a business analyst removes an association between one ormore business rules and the data mining model, the business analystraises a Model Unreferenced Event.

The business analyst communicates a Model Unreferenced Eventnotification to the business process system 102. In one embodiment, theModel Unreferenced Event notification is transmitted from the businessanalyst system 106 to the business process system 102. In oneembodiment, the business analyst system 106 invokes the web servicemethod published by the business process system 102 and transmits theModel Unreferenced Event notification via the web service invocation.

Referring to FIG. 19, a flowchart representation of one embodiment of amethod 1900 of handling a Model Unreferenced Event at the businessprocess system 102 is shown. The business process system 102 receivesthe Model Unreferenced Event notification at step 1902. The ModelUnreferenced Event notification payload includes the model identifierfor the data mining model and business rule identifiers for one or morebusiness rules that has been unreferenced by the business analyst. Thebusiness process system 102 issues a command to the model metadatarepository 108 to remove the reference to the business rule from themodel specific metadata associated with the data mining model at step1904. More specifically, the business process system 102 issues acommand to remove the business rule listing from the model business ruledependencies section.

The business process system 102 retrieves the model business ruledependencies associated with the data mining model from the modelmetadata repository 108 and determines whether there are any businessrule references to the data mining model remaining at step 1906. If thebusiness process system 102 determines that there are business rulereferences to the data mining model remaining, the method 1900 ends.

If the business process system 102 determines that there are no businessrule references to the data mining model remaining the business processsystem 102 determines whether the data mining model is a to be expireddata mining model at step 1908. Step 1908 is performed to ensure thatthe Model Unreferenced Event was not raised by a business analystresponsive to the removal of a business rule from an operational datamining model. If the business process system 102 determines that thedata mining model is not a to be expired data mining model, the method1900 ends. If the business process system 102 determines that the datamining model is a to be expired data mining model, the business processsystem 102 designates the data mining model as an expired model andraises the Model Expired Event and the method 1900 ends.

In one embodiment, once a data mining model has been designated anexpired model, the performance of the data mining model is no longerevaluated and business analysts can no longer reference the data miningmodel and/or develop business rules that are dependent on the expiredmodel. In one embodiment, the business process system 102 removes themodel specific metadata associated with the expired data mining modelfrom the model metadata repository 108 and stores the model specificmetadata in a model metadata archive repository.

Referring to FIG. 20, one embodiment of a method 2000 of managing a datamining model is shown. A first notification associated with a datamining model is received at the business process system at step 2002.Examples of the first notification include, but are not limited to, amodel created notification and a model to be expired notification.

A second notification associated with the data mining model istransmitted from the business process system 102 to the business analystsystem 106 responsive to the first notification at step 2004. Examplesof the second notification include, but are not limited to, a modelcreated notification and a model to be expired notification.

A determination is made regarding whether a first response operable todefine an association between the data mining model and business datahas been received at the business process system 102 at step 2006.Examples of a first response operable to define an association betweenthe data mining model and the business data include, but are not limitedto, a model referenced association and a model unreferenced association.

A command is issued from the business process system 102 to update themodel metadata associated with the data mining model based on thedetermination at step 2008. In one embodiment, a model metadatarepository 108 is communicatively coupled to the business process systemand is operable to store model specific metadata associated with thedata mining model. The business process system 102 updates the modelmetadata associated with the data mining model that is stored in themodel metadata repository 102.

While the steps in the method 2000 have been described in a particularorder, the steps may be performed in a different order or additionalsteps may be performed in addition to the described steps withoutdeparting from the spirit of the invention.

In one embodiment, a computer readable medium stores a computerexecutable program for managing a data mining model. The computerreadable medium includes computer readable code for receiving a firstnotification associated with a data mining model at a business processsystem, computer readable code for transmitting a second notificationassociated with the data mining model from the business process systemresponsive to the first notification, computer readable code fordetermining whether a first response operable to define an associationbetween the data mining model and business data has been received at thebusiness process system responsive to the second notification, andcomputer readable code for issuing a command from the business processsystem to update model specific metadata associated with the data miningmodel based on the determination.

It should be noted that while systems implemented using software orfirmware executed by hardware have been described above, those havingordinary skill in the art will readily recognize that the disclosedsystems could be implemented exclusively in hardware through the use ofone or more custom circuits, such as for example, application-specificintegrated circuits (ASICs) or any other suitable combination ofhardware and/or software.

The illustrations of the embodiments described herein are intended toprovide a general understanding of the structure of the variousembodiments. The illustrations are not intended to serve as a completedescription of all of the elements and features of apparatus and systemsthat utilize the structures or methods described herein. Many otherembodiments may be apparent to those of skill in the art upon reviewingthe disclosure. Other embodiments may be utilized and derived from thedisclosure, such that structural and logical substitutions and changesmay be made without departing from the scope of the disclosure.Additionally, the illustrations are merely representational and may notbe drawn to scale. Certain proportions within the illustrations may beexaggerated, while other proportions may be minimized. Accordingly, thedisclosure and the figures are to be regarded as illustrative ratherthan restrictive.

One or more embodiments of the disclosure may be referred to herein,individually and/or collectively, by the term “invention” merely forconvenience and without intending to voluntarily limit the scope of thisapplication to any particular invention or inventive concept. Moreover,although specific embodiments have been illustrated and describedherein, it should be appreciated that any subsequent arrangementdesigned to achieve the same or similar purpose may be substituted forthe specific embodiments shown. This disclosure is intended to cover anyand all subsequent adaptations or variations of various embodiments.Combinations of the above embodiments, and other embodiments notspecifically described herein, will be apparent to those of skill in theart upon reviewing the description.

The Abstract of the Disclosure is provided to comply with 37 C.F.R.§1.72(b) and is submitted with the understanding that it will not beused to interpret or limit the scope or meaning of the claims. Inaddition, in the foregoing Detailed Description, various features may begrouped together or described in a single embodiment for the purpose ofstreamlining the disclosure. This disclosure is not to be interpreted asreflecting an intention that the claimed embodiments require morefeatures than are expressly recited in each claim. Rather, as thefollowing claims reflect, inventive subject matter may be directed toless than all of the features of any of the disclosed embodiments. Thus,the following claims are incorporated into the Detailed Description,with each claim standing on its own as defining separately claimedsubject matter.

The above disclosed subject matter is to be considered illustrative, andnot restrictive, and the appended claims are intended to cover all suchmodifications, enhancements, and other embodiments which fall within thetrue spirit and scope of the present invention. Thus, to the maximumextent allowed by law, the scope of the present invention is to bedetermined by the broadest permissible interpretation of the followingclaims and their equivalents, and shall not be restricted or limited bythe foregoing detailed description.

1. A method of managing a data mining model, the method comprising:receiving a first notification associated with a data mining model at abusiness process system; transmitting a second notification associatedwith the data mining model from the business process system responsiveto the first notification; determining whether a first response operableto define an association between the data mining model and business datahas been received at the business process system responsive to thesecond notification; and issuing a command from the business processsystem to update model specific metadata associated with the data miningmodel based on the determination.
 2. The method of claim 1, whereinreceiving a first notification associated with a data mining modelcomprises receiving a first notification selected from a groupconsisting of a model created notification and a model to be expirednotification.
 3. The method of claim 1, wherein transmitting a secondnotification associated with the data mining model comprisestransmitting a second notification selected from a group consisting of amodel created notification and a model to be expired notification. 4.The method of claim 1, wherein transmitting a second notificationassociated with the data mining model comprises transmitting a secondnotification from the business process system to a business analystsystem.
 5. The method of claim 1, further comprising: retrieving atleast one model specific keyword associated with the data mining model;identifying a business analyst associated with the at least one of themodel specific keywords; and transmitting the second notificationassociated with the data mining model from the business process systemto a business analyst system associated with the identified businessanalyst.
 6. The method of claim 1, wherein determining whether a firstresponse operable to define an association between the data mining modeland business data has been received at the business process systemcomprises determining whether a first response operable to define anassociation selected from a group consisting of a model referencedassociation and a model unreferenced association has been received atthe business process system.
 7. The method of claim 1, furthercomprising: issuing a command to evaluate the data mining model;generating a model performance status associated with the data miningmodel; determining whether the model performance status is a modeldeteriorated status; and issuing a model performance notificationassociated with the data mining model from the business process systemto a model developer system based on the determination.
 8. The method ofclaim 7, further comprising receiving a third notification associatedwith the data mining model from the model developer system at thebusiness process system; making a first determination regarding whetherthe third notification is a model updated notification associated withthe data mining model; updating the model specific metadata associatedwith the data mining model based on the first determination; making asecond determination regarding whether the third notification includes amodel expiration status associated with the data mining model; andissuing a model expiration status notification from the business processsystem to a business analyst system based on the determination.
 9. Themethod of claim 1, further comprising providing a model metadatarepository operable to be communicatively coupled to the businessprocess system and operable to store model specific metadata associatedwith the data mining model.
 10. The method of claim 9, wherein providinga model metadata repository comprises providing a model metadatarepository operable to store model specific metadata selected from agroup consisting of a basic model information, a model schemadefinition, a model assumption, a model specific keyword, a modelperformance evaluation definition, a model performance event trigger, amodel business rule dependency, and an inter-model dependency.
 11. Acomputer readable medium for storing a computer executable program formanaging a data mining model, the computer readable medium comprising:computer readable code for receiving a first notification associatedwith a data mining model at a business process system; computer readablecode for transmitting a second notification associated with the datamining model from the business process system responsive to the firstnotification; computer readable code for determining whether a firstresponse operable to define an association between the data mining modeland business data has been received at the business process systemresponsive to the second notification; and computer readable code forissuing a command from the business process system to update modelspecific metadata associated with the data mining model based on thedetermination.
 12. The computer readable medium of claim 11, wherein thecomputer readable code for receiving a first notification associatedwith a data mining model comprises computer readable code for receivinga first notification selected from a group consisting of a model creatednotification, and a model to be expired notification.
 13. The computerreadable medium of claim 11, wherein the computer readable code fortransmitting a second notification associated with the data mining modelcomprises computer readable code for transmitting a second notificationselected from a group consisting of a model created notification and amodel to be expired notification.
 14. The computer readable medium ofclaim 11, wherein the computer readable code for transmitting a secondnotification associated with the data mining model comprises computerreadable code for transmitting a second notification from the businessprocess system to a business analyst system.
 15. The computer readablemedium of claim 11, further comprising: computer readable code forretrieving at least one model specific keyword associated with the datamining model; computer readable code for identifying a business analystassociated with the at least one of the model specific keywords; andcomputer readable code for transmitting the second notificationassociated with the data mining model from the business process systemto a business analyst system associated with the identified businessanalyst.
 16. The computer readable medium of claim 11, wherein thecomputer readable code for determining whether a first response operableto define an association between the data mining model and business datahas been received at the business process system comprises computerreadable code for determining whether a first response operable todefine an association selected from a group consisting of a modelreferenced association and a model unreferenced association has beenreceived at the business process system.
 17. The computer readablemedium of claim 11, further comprising: computer readable code forissuing a command to evaluate the data mining model; computer readablecode for generating a model performance status associated with the datamining model; computer readable code for determining whether the modelperformance status is a model deteriorated status; and computer readablecode for issuing a model performance notification associated with thedata mining model from the business process system to a model developersystem based on the determination.
 18. The computer readable medium ofclaim 17, further comprising computer readable code for receiving athird notification associated with the data mining model from the modeldeveloper system at the business process system; computer readable codefor making a first determination regarding whether the thirdnotification is a model updated notification associated with the datamining model; computer readable code for updating the model specificmetadata associated with the data mining model based on the firstdetermination; computer readable code for making a second determinationregarding whether the third notification includes a model expirationstatus associated with the data mining model; and computer readable codefor issuing a model expiration status notification from the businessprocess system to a business analyst system based on the determination.19. The computer readable medium of claim 11, further comprisingcomputer readable code for storing model specific metadata associatedwith the data mining model at a model metadata repository, the modelmetadata repository being communicatively coupled to the businessprocess system.
 20. The computer readable medium of claim 19, whereinthe computer readable code for storing model specific metadataassociated with the data mining model at a model metadata repositorycomprises storing model specific metadata selected from a groupconsisting of a basic model information, a model schema definition, amodel assumption, a model specific keyword, a model performanceevaluation definition, a model performance event trigger, a modelbusiness rule dependency, and an inter-model dependency.
 21. A systemfor managing a data mining model, the method comprising: means forreceiving a first notification associated with a data mining model at abusiness process system; means for a data mining model management modulefor transmitting a second notification associated with the data miningmodel from the business process system responsive to the firstnotification; means for a data mining model management module fordetermining whether a first response operable to define an associationbetween the data mining model and business data has been received at thebusiness process system responsive to the second notification; and meansfor a data mining model management module for issuing a command from thebusiness process system to update model metadata associated with thedata mining model based on the determination.